Skip to content

[SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates#57223

Open
stevomitric wants to merge 2 commits into
apache:masterfrom
stevomitric:stevomitric/spark-57811-nanos
Open

[SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates#57223
stevomitric wants to merge 2 commits into
apache:masterfrom
stevomitric:stevomitric/spark-57811-nanos

Conversation

@stevomitric

@stevomitric stevomitric commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Add non-ANSI type-coercion arms so that a string compared against a nanosecond-precision timestamp column (TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7, 9]) is cast to that nanos type, mirroring the existing microsecond TimestampType handling (StringPromotionTypeCoercion equality arms + TypeCoercion.findCommonTypeForBinaryComparison).

Why are the changes needed?

Micros TimestampType has string-coercion arms that honor ...datetimeToString.enabled (legacy → promote to string; equality → cast to timestamp). Nanos had none, so it fell through to config-blind AtomicType promotion. This adds the arms so nanos matches TimestampType's legacy behavior.

Does this PR introduce any user-facing change?

Only under legacy datetimeToString=true + ANSI off: range comparisons (<, BETWEEN, …) now promote to string (matching micros); equality unchanged. All other configs identical.

How was this patch tested?

Extended existing suites and added golden file tests.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code

@stevomitric

Copy link
Copy Markdown
Contributor Author

cc @cloud-fan, @MaxGekk PTAL.

@stevomitric stevomitric changed the title [WIP][SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates [SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates Jul 16, 2026
…oercion in comparisons and predicates

Type coercion previously handled only microsecond TimestampType when one
operand of a comparison or predicate is a string: the string was cast to
TimestampType so subsecond digits compared exactly. Nanosecond timestamp
types (Timestamp{LTZ,NTZ}NanosType) had no such arm, so a string compared
against a nanos column fell through to generic handling and lost the 7th-9th
fractional digits.

This adds the missing non-ANSI arms, mirroring the existing microsecond
handling: two Equality arms in StringPromotionTypeCoercion cast the string to
the concrete nanos operand type, and two findCommonTypeForBinaryComparison
cases in TypeCoercion return that nanos type for the range path. The concrete
family and precision of the timestamp operand are preserved, the arms are
ordered so equality fires before the range path, and legacy
castDatetimeToString is honored (range promotes both sides to string, equality
still casts the string to nanos). ANSI type coercion is intentionally left
unchanged.

Tests: TypeCoercionSuite "binary comparison with string promotion" gains
nanos assertions (all precisions, both families, plus a legacy-mode block that
exercises the new arms); TimestampNanosWideningSuiteBase adds an end-to-end
string-operand test; and the timestamp-{ltz,ntz}-nanos golden files add
equality/range/BETWEEN cases.

Co-authored-by: Isaac
@stevomitric stevomitric force-pushed the stevomitric/spark-57811-nanos branch from 35428ec to 660d8d1 Compare July 16, 2026 13:02
@cloud-fan

Copy link
Copy Markdown
Contributor

can you fix merge conflicts?

…7811-nanos

# Conflicts:
#	sql/core/src/test/resources/sql-tests/analyzer-results/timestamp-ltz-nanos.sql.out
#	sql/core/src/test/resources/sql-tests/analyzer-results/timestamp-ntz-nanos.sql.out
#	sql/core/src/test/resources/sql-tests/inputs/timestamp-ltz-nanos.sql
#	sql/core/src/test/resources/sql-tests/inputs/timestamp-ntz-nanos.sql
#	sql/core/src/test/resources/sql-tests/results/timestamp-ltz-nanos.sql.out
#	sql/core/src/test/resources/sql-tests/results/timestamp-ntz-nanos.sql.out
@stevomitric

Copy link
Copy Markdown
Contributor Author

can you fix merge conflicts?

done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants